Repository navigation
fix(batch): report short-circuited FIFO records that don't match the model - #8663
Conversation
92dc822 to
1545ba0
Compare
|
Hi @soujasK, I saw you closed this one. No problem, I just want to understand what you'd prefer. The bug is real and triaged for 3.40.0, and I already worked on it on my side. Do you want to reopen so I review your PR, or should I open a new one myself? Either way is fine, just let me know. |
|
Reopening this. I closed it by mistake. I need a bit more time to test and validate it properly, so I'll update here once that's done. |
|
please review mine |
leandrodamascena
left a comment
There was a problem hiding this comment.
Thanks @soujasK, and sorry for the wait. I ran this on top of today's develop in a clean environment: the batch suite, the required-dependencies session, ruff, mypy and xenon, all green. The change is exactly what the issue needed: a record that does not match the model after a short circuit becomes one more entry in batchItemFailures instead of failing the invocation. This goes out in 3.40.0, no rebase needed, credit in the release notes.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #8663 +/- ##
========================================
Coverage 97.18% 97.18%
========================================
Files 323 323
Lines 16542 16546 +4
Branches 1495 1495
========================================
+ Hits 16076 16080 +4
Misses 326 326
Partials 140 140 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|



Issue number: closes #8661
Summary
Changes
Once a record fails,
SqsFifoPartialProcessorshort-circuits the remaining records (or the rest of the failed group withskip_group_on_error=True) and converts them with the customer's model before passing them tofailure_handler. That conversion was outside anytry, so a record that didn't match the model raised out ofprocess()and failed the whole invocation.Short-circuited records never reach the record handler, so if the model conversion fails they're now registered as event source data classes, like other poison pills. Records that do match the model are still registered as model instances, so
fail_messagesdoesn't change for them.Tests: a batch with a successful record, a failing record, a record that doesn't match the model in the same group, and a successful record in another group, parametrized over
skip_group_on_error. Both cases raisedValidationErrorbefore the change and pass after it.tests/functional/batchpasses on Linux (Python 3.12), along with ruff, mypy, ty, bandit and xenon.User experience
Before: a malformed record after a failure fails the whole invocation, so every record is retried, including already processed ones (and, with
skip_group_on_error=True, the successful records of other message groups).After: the malformed record is reported in
batchItemFailuresalong with the other short-circuited records.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.